I like the features of this PR.

I wonder whether we can consolidate some naming consistency around the behaviors of:

• contents: c['x'].contents returns the names of the Union's data_sources
• data_sources(): c['x'].data_sources() returns file-level details of the data_sources backing the Union's members
• data_source: URL parameter containing the name of a Union member data_source, as in PUT /api/v1/table/full/x?data_source=table

On the other hand, this might already be as simple as it gets, and I just need a minute to get comfortable with the usage. :)

• contents: c['x'].contents returns the names of the Union's data_sources

This is perhaps a bit too reductive a statement, as really this contains structure and named sources and can iterate through the members.

Even though the contents map 1:1 to a data_source, it might be cleaner to not use them interchangeably. I will refer to the contents here as parts for brevity and to avoid possible confusion with Container. I'm not attached to the name parts.

Then the above could be used something like this...maybe?

client['x'].parts   # <UnionParts {'table', 'C'}>
client['x'].data_sources()   # low-level storage details, encoded in DataSource and Asset
client['x']['C'].data_sources()   # one-element list, or would data_source() be better?

PUT /api/v1/array/full/x?part=C ...BODY  # Refer to the union member, rather than its data source

I like that suggestion very well, and I like the name part.

In the future (months away) I hazily foresee enabling Tiled to track multiple versions of data:

• replicas stored "on prem" or in the cloud
• copies with different file formats and/or chunking to make a range of use cases fast

This is why data_sources() is a one-element list, in anticipation of there being more than one someday, and wanting to leave room for that.

But, this also underlines why separate part and data_source could be important: they happen to be 1:1 today but may not always be.

I also like that giving a distinct name to this concept helps clarify which abstraction level you are operating at. Referring to a part moves you from (1) to (2) but until you mention a data_source you have not crossed into (3).

Rebased on main after merging #661. The renaming of data_source to part, in the places discussed has been done.

The to-dos...

Implement read() on UnionClient (i.e. c['x']) itself, which could pull each data source in turn and return an xarray.Dataset.
Implement GET /union/full/{path} to enable bulk download. This will work similar to container export.

I would at least like the validate this branch by connecting it to be a Bluesky document stream before merging.

are, I believe, strictly additive and could be done in separate PRs or in this PR.

Rethinking this in terms of a "view" is very interesting. Off the top of my head, I like that:

• A "view" is a widely-recognized concept and may require less explanation.
• This seems like it might address a separate issue noticed by @genematx, that in the current data model data and timestamps are allowed to be uneven in length. (And, in fact, in the middle of an update, they always will be.) If there were one table with data and timestamps "views" into subsets of it, that would fix the problem.

For example, maybe this is what a event stream could look like. Notice that:

• The layout and URLs are backward compatible with what we have been doing. It simply adds a new key, __name_me__ (needs a good name...).
• It ensures that data and timestamps are the same length because they are views on the same table.
• It exposes the "real" structures to the client, but gives a flattened view of it too.

primary
├── data  # view
│   ├── time
│   ├── motor_readback
│   ├── motor_setpoint
│   └── image_array
├── timestamps  # view
│   ├── time
│   ├── motor_readback
│   ├── motor_setpoint
│   └── image_array
├── __name_me__
│   ├── event_table  # values stream inline in Event documents
│   │   ├── time
│   │   ├── data_motor_readback
│   │   ├── data_motor_setpoint
│   │   ├── timestamps_motor_readback
│   │   └──  timestamps_motor_setpoint
│   └── image_array  # externally-written array data

Either path we take, union or view, would be speculative. We have to add this and really try it to understand how it flies. There is some risk either way.

View, in addition to being a more widely recognized concept, could solve a broader category of problems for us. BlueskyRun is very nested, and this can mean a lot of clicks in the UI. I can imagine a view (or views) of the good parts, flat. Views could be combined with containers to create a nested-but-not-THAT-nested structure. (Dare I call it a "projection"?)

I think we should keep the specification light and focused on current requirements, but I can see a lot of useful scope in this direction. Maybe I’ll start by opening a parallel PR for comparison that builds on this branch but refactors union into view.

This is why we keep @dylanmcreynolds around. :-)

Could views be a better place to put Databroker's projections and projectors? To sum up, projection are a way add a mapping to the start document. projectors are a python function that take a run, its projection and returns an xarray with datafields mapped as specified in the projection. The major idea was to have a simple way to create multiple views from the same run. One projection could be customized for a user interface, another could be customized for a particular ontology (like nexus).

If we took this view idea even further, the definition of the view could also include information about mapping to a desired ontology.

I think we should keep the specification light and focused on current requirements,

That sounds prudent.

... but I can see a lot of useful scope in this direction.

Do you envision that views might evolve to include keys from other nodes (ancestors, siblings, 2nd-cousin-once-removed), or is that something that should be firmly disallowed? I can imagine complications arising from access policy as well as latency/timeouts from trying to include too many keys.

I think there will be significant pressure to enable views that reach anywhere across the tree:

• Mix raw data and analyzed
• Experiment with alternative views outside of the main tree without making it "noisy"
• Probably more….

The specification of a union node involves listing data sources directly. If the specification of a view involves instead referencing other nodes, I think that access control is manageable, and the scaling can be managed if we enforce reasonable limits on the number of nodes allowed in one view.

It ensures that data and timestamps are the same length because they are views on the same table.

More generally, would merged views only work if all parts have the same length (number of rows)?...and if so would it enforce that by:

• rejecting data sources that don't meet that condition? --OR--
• returning a table with number of rows equals to the shortest part (like python's zip())? --OR--
• returning a table with number of rows equals to the longest part (like python's itertools.zip_longest())? Filler value could be None, numpy.nan, "", or probably a user-supplied value --OR--
• any of the above, depending on a query parameter passed by the caller?

Or should views require a key to join upon, using merge behavior such as LEFT OUTER JOIN, RIGHT OUTER JOIN, FULL OUTER JOIN, INNER JOIN?

Could views be a better place to put Databroker's projections and projectors?

So rather than injecting that info into the run documents, you're suggesting to instead let tiled handle that when the data gets accessed? That makes a lot of sense.

Run documents would be more "pure"--less coupled to how someone thought they should be viewed when they were recorded. When new views get dreamed up, they could be added to the tiled server config--restarting the server (or maybe registering the new view with the catalog) would allow that view to be applied to all data new and old.

So rather than injecting that info into the run documents, you're suggesting to instead let tiled handle that when the data gets accessed? That makes a lot of sense.

Maybe. The projections schema was added to the run start document so that they could be the default projection for a particular run. If a newer version were available, the projector code could use it if asked to.

But I only know of one case where projection/projectors were used since they were developed four years ago. Maybe that's a sign? Perhaps the issue is they weren't needed much, perhaps they weren't advertised well, or perhaps the mechanism was too complicated.

I think there will be significant pressure to enable views that reach anywhere across the tree.

That's an interesting thought. I feel like that if it kept the scope in check, I'd be happy to say that a view was limited to objects of the same row/timestamp. We could call it RowView and if we decided we need something more flexible in the future, come up with a SuperView with extra powers?

To fit our present use case, we would need a view to look a lot like union, mixing tables and arrays in one namespace. There would be [edit: NO] special guarantees about the relation between the items in the namespace, nothing about their length or how to join them. (It goes without saying that the constitutive tables would each internally have the normal length guarantee that it is made of whole rows.)

I think the change from union is, the parts in a view would have their own canonical locations in the tree, as normal nodes. A view becomes an additional place to get (mixtures of…) nodes. Each views look a lot like a union would have, but their parts are pointers to first-class nodes, not to captive data sources. This enables us to separately build multiple views on the same data. And it avoids placing the canonical data in a weird structure that would require explanation (union).

As far as "projections" goes, I like that this presents clients with the result rather than instructions (in a novel descriptive language…) for rearranging a structure on the client side.

Yes, one can imagine constructing views dynamically through special adapters added to the server config—I think this is what @padraic-shafer’s last message envisions. For our present requirements I would start, though, by building them as static entities. The client specifies, via a HTTP request, "Add a view node to the database that combines array node X and columns A, B from table node Y into one namespace and present it to clients.

The client specifies, via a HTTP request, "Add a view node to the database that combines array node X and columns A, B from table node Y into one namespace and present it to clients.

I think this Union PR didn't yet add the capability to read the combined result at once, right? So what do we think should happen when the outer dimension of array X differs from the length of table Y? (or equivalently when event_table has more rows than image_array has images in the earlier example?)

I might be quibbling about edge cases. But I wonder about how 'edge'y these cases are. We could of course enforce this when the view node (meaning the outer node, not the projection node) is created, and then wait to see if it runs into issues during testing.

That's an interesting thought. I feel like that if it kept the scope in check, I'd be happy to say that a view was limited to objects of the same row/timestamp. We could call it RowView and if we decided we need something more flexible in the future, come up with a SuperView with extra powers?

For our present requirements we can keep this pretty limited, and spin out further discussion on whether and how to expand it once we have something to play with.